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ABSTRACT 



Evaluation of instructional programs involves the 
process of ascertaining the decisions to be made. The decision maker, 
not the evaluator, determines the guestions to be asked or the 
decisions to be made. A second task of evaluation deals with the 
specification of reguired information in light of system objectives. 
The task of the evaluator includes the development of research design 
and of instruments to provide the information appropriate to the 
decisions that must be made. The five stages of evaluation are; (1) 
needs assessment, (2) Planning, (3) program implementation, (4) 
program improvement, and (5) program certification. Each stage is 
discussed and the categories of decisions are related to these stages 
to demonstrate the relevance or lack of relevance of students* 
behavioral objectives in each stage. (ME) 
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The Use of Behavioral Objectives in Evaluation: 
Relevant or Irrelevant? 

The question posed by the title of this paper is 
not readily answerable. Indeed, there is no single solution 
to the question. The use of specified behavioral objectives 
in evaluation is neither relevant nor irrelevant. It is 
the threefold thesis of this paper that (l) behaviorally 
stated objectives are of relevance only to certain stages 
in the evaluation process; (2) even in those stages where 
it is relevant to state student behavioral objectives, 
objectives specification alone ceases to be of singular 
significance with the increasing complexity of the program; 
and (3) even in relaitively non-complex programs within 
stages amenable to objectives specification, there is 
little research evidence showing whether evaluation using 
specified student behavioral objectives "makes a difference. 

The intent of this paper, however, is not to discount 
completely the value of specifying objectives in the 
evaluation of instructional programs; to do so would be 
ludicrous. The need for specifying objectives was pointed 
out by Lewi.s Carroll in Aliceas Adventures in Wonderland . 



in the following conversation between Alice and the 



’’Come, it»s please so far,'' thought Alice, 
and she went on. "Would you tell, please, 
which way I ought to go from here?" 

"That depends a great deal on where you 
wajit to get to," said the cat. 

"I don*t much care where..." said Alice. 



"Then it doe.sn»t matter which way you go," 
said the cat. 

Behavioral objectives specification is not necessarily 
a panacea for evaluation problems of all types. While 
all enterprises should have a goal — ^^these goals are not 
necessarily always specifiable in student behavioral terms. 
Moreover, when complex systems are involved the specifica- 
tion of objectives in behavioral terms has been somewhat 
overplayed in its relative importance. I would submit 
also that the broadening definition of evaluation has 
considerably modified views about the need for specification 
of behavioral objectives. 



Introduction 

The last two years have represented an exciting 
period in the field of evaluation. Indeed, it would not 
be an overstatement to maintain that evaluation as a field 
has just begun to assume an identity of its own. 

I would agree i/ith Egon Guba that a major 
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failing of evaluation today stems from the lack of an 
adequate definition. Past definitions have equated it 
with either: (l) measurement and testing, (2) statements 

of congruence between performance .and objectives, or 
(3) professional judgment. None of these by itself is 
really an inclusive enough', definition for the multiplicity 
of activities now regarded as evaluation. During the 
past year, a consensus has been developing concerning a 
broader, more comprehensive definition of evaluation. 

This expanded viev/ takes into consideration the decision 
making functions, ance an evaluation must be 
predicated on, and adapted to, the specific problem or 
situation under analysis. 



Definition 

In view of the fact that there is no definitive 
statement of evaluation, it would be inappropriate and 
inaccurate of me to present my definition as ’’the” generally 
accepted one. However, 

in an effort to provide some framework for this paper, 

I will, somewhat hesitantly, step forward and present 
my definition of eva?i.uation. . . . Evaluation is the pro_ces^3 



of ascertaining the decisions to be made, selecting 



related information, and collecting and analyzing that 
infornfiation in order to report summary data useful to 
decision makers in selecting among alternatives . 

The first part of the definition of evaluation pre- 
sented here deals with ascertaining the decisions to be 
made. The decision maker determines the questions to be 
asked or the decisions to be made and not the evaluator. 

The task of the evaluator is to deterirdrie from the decision 
maker the decisions for which information is required. 

The evaluator can and should, however, point out inconsist- 
encies, potential difficulties, or additional data that 
might modify the decision makers views on the relevance 
of certain decisions. 

A second task of evaluation deals with the specifi- 
cation of required information in light of system objec- 
tives. The specific nature of the information required 
will differ, depending on the kind of decision to be made. 
If the decision relates to specifying the needs of a 
community, the information requirements will be quite 
different from those of a study on the comparable success 
of two programs under experimental conditions. The task 
of the evaluate]? in specifying information requirements 
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includes the development of the resesvrch design of the 
project, and the selection and/or development of instruments 
designed to provide the information appropriat-e to the 
decisions which must be made<. 

Data collection and analysis are tasks of prime concern 
bo the evaluator. The evaluator will encounter different 
problems associated with these tasks, depending upon the 
nature of the unit being evaluated, the kind of question 
being asked and other things. 

One of the most vital parts of the evaluation process 
is reporting to the decision maker. Most evaluators 
often overlook this function as being merely a pro forma 
exercise. Indeed, if the purpose of evaluation is to 
provide infornation that will enable decision makers to 
form judgments about a program or about alteiviatives, 
then the nature and form of the reporting should be 
appropriate to the problem and the audience. 



Stages of Evaluation 



This definition of evaluation carries with it 
a concern for the decisions to be made. Thus, if 
we are to understand the evaluation process, it is 
necessary to categorize educational decision situations. 

Tn this classification, it would be necessor^ to 
examine the nature and kinds of decisions that are 
likely to require evaluative dstta. 

I have identified what I consider to be the five 
stages of an evaluation — each is designed to provide 
and report information useful to a decision maker in 
making judgments. They are: (l) needs assessment, 

( 2 ) .planning, (3) program implementation, (4) program 
improvement, and (5) program certification. I should 
note that I have borrowed liberally in the development 
of these stages from the work of Dan Stufflebeam, Bob 
Stake, Mai Provus and others. 

The first area in which evaluation might take place 
is in the assessment of needs. Needs assessment is a 
means of determining the educational objectives most 
appropriate for a particular situation. The needs may 
be represented as the gap between the goal and the present 
state of affairs. Thus, the evaluation probiLem becomes 



one of assessing the needs of students, of the community, 
and of society in relation to the existing situation. 

Needs assessment does not refer to specification of process 
characteristics appropriate for a district, school, or 
classroom. The needs assessment must be related to the 
ultimate behavior of clients of one type or another (Pupils, 
parents, commvmity, etc.; all are clients of the school). 

To put it simply, needs assessment must be a statement of 
objectives in terms of outputs rather than process charac- 
teristics of the system. 

It is obvious from these examples as well as to all 
of those who have been engaged in needs assessment under 
a Title III program, that the process of deciding the 
purposes of needs assessments, as well as specifying, 
collecting, analyzing, and reporting information is quite 
different from the methodology and techniques ordinarily 
associated with typical evaluation. 

Planning 

The planning stage in evaluation is concerned with 
information which will enable the decision maker to select 
between alternative processes in order to make a jiudgment 
as to which process should be introduced into the system 
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in order to fill raost efficiently the critical needs which 
have been previously determined. After the decision maker 
receives the needs assessment evaluation, he might make 
a decision as to the appropriate means of fulfilling that 
need. Alternatively, he might designate several possibilities 
and ask the evaluator to provide information on the possible 
iiapact of each,, Thus, in the planning stage, the evaluator 
provides the data for an evaluation of a program prior 
to its inception. The task of the evaluator is to look 
forward to the attainment of goals cind to determine the 
likely goal achievement or outcomes. To repeat this, in 
yet another way, the purpose of evaluation in the planning 
stage is to assess the potential relative effectiveness 
of different courses of action. 

It is quite obvious from this discussion that vhe 
collection and analysis of data of the type required for 
this evaJl>uation stage will be quite different from collection 
and analysis problems for other stages. The techniques 
may require both internal and external evaluation criteria. 
(The most appropriate technique might be informed judgment 
or other so called soft data. ) 



Program Implementation 



The next step in the evaluation process is determining 
the extent to which the program has been implemented in 
the manner in which it was described in the design. (A 
part of the information specification, collection, analysis 
and reporting process is the specification of the design 
or procedures by which each of these activities will be 
accomplished, ) 

In the case of an existing program, where no known 
changes have been implemented, the evaluation task for 
this stage is to determine the degree to which planning 
descriptions of the program coincide with the actual 
program. 

Program Improvement 

The evaluator can assume a leadership role in program 
improvement by providing as much information as possible 
about the relative success of its parts. In order to 
perform program improvement evaluation, it is necessary 
to recognize the basically interventionist role that the 
evalfiiator has been asked to play. As the evaSluato.r 
identifies problems and collects and analyzes information, 
data is presented immediately to the decision-maker in 
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order that changes to improve the operation of the 
program might be executed within the system. This 
stage of evaluation has often been overlooked or ignored 
by the traditional evaluator who has attempted to reproduce 
the antiseptic sterility of a laboratory in the real 
world. This approach may make a fine experiment, but it 
does little to improve a program which is often not in 
final form. 

Program Certification 

Finally, evaluation must provide information to the 
decision maker that will enable him to make judgment about 
the instructional program as a whole. This is the "audit" 
stage of evaluation. The evaluator might attempt to 
provide information which will enable the decision maker 
to determine whether the program should be eliminated, 
modified, retained or expanded. 

In this stage, the need for valid and reliable data 
would generally mandate that the evaluator attempt to 
apply as rigid a set of controls as possible. The 
evaluator might use pre and post test designs, and emplpy 
sophisticated statistical techniques for analyzing the 
data. Whenever possible, intervention should be avoided 
in this stage. 



Use of Student Behavioral Objectives in Various 



Evaluation Stages , 

I will discuss each of the stages and the categories 
of decisions related to these stages in order to demonstrate 
the relevance or lack of relevance of students » behavioral 
objectives for each. A decision is associated with each 
of the five stages and it is the job of the evaluator 
to provide the information that will assist the decision 
maker in selecting between alternatives for that decision. 
The nature of the decision at each stage, I believe will 
demonstrate, that information on the achievement of student 
behavioral objectives is not relevant to some. stages and 
is not the only source of information appropriate for other 
stages. 

In the discussion that follows I do not mean to imply 
that the evaluator will necessarily participate in each 
stage of the evaluation. In some instances prior decisions 
may already have been made and the evaluator may be asked, 
simply, to provide information for succeeding stages. In 
other instances the nature of the information to be col- 
lected may be relatively simple and the process cf informa- 
tion selection, collection and analysis may be internalized 
by the decision maker or his staff. However, for the sake 



of clarity, vre will assume a hypothetical situation where 
the evaluator is asked to provide information for decisions 
at each of the five stages. 

The first question facing the evaluator is related to 
selection of objectives for the system or modification of 
existing objectives. Thus, depending upon the situation, 
the decision maker may want information on whether various 
constituent bodies (i.e., the community) concur w5.th the 
existing objectives of the system and what changes are 
needed. It may be appropriate to present information on 
the potential relevance of alternative objectives in terms 
of possible future significance. 

In a h^jrpothetical situation, a school principal might 
be faced with a budgetary decision and want to get some in- 
sights as to how best to spend an incremental budget amount. 
He is anxious to spend this in a manner that is likely to 
be most beneficial to the school in terms of its needs. 

The evaluator has been asked to provide information about 
various possible objectives for the system, including some 
presently stated objectives which may be inadequately met. 

Thus, the evaluator iriay inform the decision maker that 
a number of behavioral objectives of the system have been 
considered highly relevant by the community, that the 
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evidence appears to demonstrate that these have been inade- 
quately met. High on this list mi,ght be the student »s 
inability to defend themselves, i.e., trained in "the 
art of self defense,” against attacks by other students. 

He might also provide infoimtion in which he attempts to 
indicate the potential value of selecting ”self defense” 
as an objective of the system. Thus, the needs assessment 
evaluation would provide the decision maker with informa- 
tion that would assist him in selecting between alterna- 
tive objectives. The information is provided by the evalua- 
tor, but the relative weightings of the alternatives must 
be made by the decision maker. 

It is obvious from this example that the major source 
of information provided by the evaluator in this stage is 
related to student »s behavioral objectives for the system. 
The evaluator, in essence, provides alternative objectives 
along with other descriptive information to the decision 
maker. 

In the planning stage, the evaluator provides infor- 
mation about possible means of achieving the objectives. 

The question asked by the decision maker is ”what process 
is to be chosen from among a list of alternatives?” 

The evaluator is not an instructional development expert 
and ordinarily should not assume the job of developing 
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a program appropriate to the stabed objective. However, 
the decision maker might have narrowed his choice to se- 
veral alternatives and would like additional information 
on each of these alternatives. 

In the case previously presented, if we assume 
that the decision maker has selected a behavioral objec- 
tive related to ”self defense” instruction and has con- 
sidered three alternative processes, then the evaluator 
inight provide information related to each of these processes. 
The information of necessity will be limited in this pre- 
implementation stage. The evaluator, of course, will examine 
each of the processes in terms of various internal criteria, 
such as the extent to which material purports to achieve 
the specified objective, the clarity of the materials, 
the cost of the materials, etc. 

In addition, the evaluator may invoke certain exter- 
nal criteria. An examination might be made of the litera- 
ture related to the use of this process to determine the 
extent to which it had been found to be successful in 
similar situations. In the absence of any evidence related 
to the use of these materials, the evaluator might choose 
to use systematically sampled expert judgment about the 
potential worth of each of the processes. Thus, given 
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the information collected and analyzed for this stage, 
the decision-maker would be in a position to make a more 
rational choice. In this phase, the provision of infor- 
mation centers about data related to alternative processes. 
While it is true that the processes are examined in rela- 
tion to potential desired student outputs, the main source 
of information for the second stage of evaluation is not 
information on student behavioral objectives but on processes. 

The eva3-uation related to the 3rd stage, program 
implementation, has as its purpose providing information on 
whether the process which was selected has been jjuplemented 
according to plan and whether the context of the situation 
in terms of the fitted attributes of the program have been 
described properly in the p3, arming stage. That is, did the 
equipment arrive on time? Does the description of the 
students in the planning stage and which was considered at 
the time when the process was selected, correspond with the 
actual situation? It is obvious that in this stage, also, 
specification of student behavioral objectives is not 
of critical importance. 

In the example that we have been using, let us as- 
sume that the decision maker has examined the alternative 
processes and has decided to introduce a course in shot- 
gun manufacturing to achieve the objective related to 
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”self defense.” (3ne question for the evaluator is: did the 
gun barrels arrive on time? 

In the 4th stag<3, program, improvement, specified student 
behavioral objecjtives are of major importance. In this 
stage, the evaluator is concerned with determining changes 
in students suid obsenring students^ achievement on a regu- 
lar basis in order to provide feedback to the decision 
maker that would be helpful in modifying the program. In 
addition to information related to the achievement of 
students on certialn objective dimensions, the evaluator 
has as his function wi.thin this stage the provision of 
information relating to the efface of the introduced 
process upon other processes of the system. Thus, in 
the example w'cj have been using, the evaluator might note 
that while students seem to be doing very well in leaiviing 
to constl*uct shotguns, there appear to deleterious effects 
upon teacher-student relationships. Moreover, other stu- 
dents in the school seem., for some reason or another, to 
be afraid of those in the experimentsil program. Finally 
the evaluator might observe that the general appearance 
of the school building has suffered. (The walls are 
pitted, and indeed many have large gaping holes in them). 

On the basis of this Information, the decision maker may 
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chooise to modify the program, expand it because of the 
surprisingly good results, or perhaps even delete it im- 
mediately. 

Let us assume that the program has been allowed to 
contin\ie and has gone through the program improvement 
stage to the point where the decision maker is now satis- 
fied i^th the program and wants to provide a rigid em- 
pirical test. At this juncture, the evaluator may be 
called upon to provide an evaluation related to the pro- 
gram certification function. The evaluator is not being 
asked to certify the program but rather to provide in- 
formation that will allow a decision to be made about 
certification. As opposed to the previous stages, the 
role of the evaluator in program certification is non- 
interventionist. Thus, in the example noted above, the 
evaluator will attempt to provide information to the de- 
cision maker on final (or nearly final) outputs of the sys- 
tem in student or other terms as a function of the course in 
shotgun manufacturing. Again, student behavioral objec- 
tives should be considered. The evaluator will also want 
to provide information on the extent to which students are 
now better able to defend themselves. There are, however, 
a number of other outcomes of the system that were perhaps 
not anticipated which might well be reported to the decision 
maker as a part of the program certification 
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evaluation. For example, he might note there has been 
a considerable increase in the amount of violence in the 

community, an increase in the number of armed robberies, 
etc, 

I have attempted to demonstrate in the preceding 
paragraphs that behavioral objectives are of considerable 
relevance to various stages of evaluation, are of rele- 
vance along with other kinds of information in several 
stages of the evaluation and of little relevance and, 
indeed, irrelevant in other stages. 

System Comnlexitv 

In areas traditionally conceived of as evaluation 

i.e., program improvement and program certification— there 
is ordinarily a great need for specifying objectives in 
behavioral terras, but even here I must sound a dissident 
note. Those advocating the use of behavioral objectives 
as the main basis for evaluation are usually concerned only 
with the individual student, or at most, the classroom, as 
the unit of analysis. The examination of more complex 
programs often makes it impossible to state behavioral 
objectives at the outset. One can think of broad scale 
educational systems 
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wit h outcomes that are not clearly definable and where 
the process of specifying objectives is an iterative 
one. The complexities of this kind of system are often 
so great that to speak of objectives in any concrete 
sense is to mask the real outputs of the system. The out- 
comes and consequences of all of the many interactions 
within a system are very great, and are often at consider- 
able variance with the objectives of the system. 

Also, the nature of the context at this macro-level 
of complexity is of considerable significance. While we 
would maintain that at the micro-level, the most important 
element in evaluation is ohe specification of objectives, 
in large educational systems, the context or nature of the 
surroundings has tremendous impact on the outcomes of the 
system. The Coleman Report is just one example of a whole 
line of research which has tended to substantiate this. 

Other difficulties of evaluating complex systems in- 
volve accurate specification of the instructional treatment. 
That is to say, often the instructional treatment is nedther 
clean, easily identifiable, nor easily reproducible. It 
is, instead, a vast array of complex, interactive elements 
loosely called instruction. 

Thus, we have shown that what is required in this 
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kind of evaluation is not simply a specification of 
objectives, but rather a total examination of a system, 
with all of the implications that derive from systems 
theory. A systems evaluation carries with it the 
necessity for specifying the inputs and outputs of the 
system, and the understanding that the process of evalu- 
ation must be an iterative one in which successive stages 
produce additional information. 

If we think of evaluation as being the process of 
selecting, collecting, analyzing, and providing infor- 
mation for decision makers, then the implications of the 
data requirements for the evaluation of complex educa- 
tional systems are readily apparent. In addition to 
specifying the objectives of the system and the degree 
to which the system has met these objectives, data must 
also be provided on other outcomes (unanticipated outcomes, 
consequences), on the inputs, on accurate descriptions of 
the alternative processes used, and on the input-output 
relationships, especially as they relate to the factors 
which can be considered by the decision maker. 

An activity currently underway at the UCLA R & D 
Center is designed to provide answers to questions about 
the appropriate information necessary for various decisions. 
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The 1 ^ 70 ject, the School Evaluation Project, is being 
directed by Steve Klein and myself and is attempting 
to develop an information system that will help school 
principals predict student outputs of their schools and 
make decisions about how to improve these outputs. 

The Project is uniquely different from most socio- 
psychological descriptive studies of education in that 
the orientation focuses on the decisions made by school 
principals. The project will attempt to determine infor- 
mation requirements (that is, appropriate evaluations) 
for a number of decisions or classes of decisions. It is 
hoped that the results of this research will provide insights 
into the relative importance of various kinds of information, 
including those related to behavioral objectives, for 
various types of educational decisions. 

Lack of Research Evidence 

Finally, it is imperative to note that even for re- 
latively discrete units of evaluation, there is not de- 
finitive evidence that behavioral objectives specifica- 
tion "makes a difference.” It has not been substantiated 
clearly that specifying objectives in behavioral terms 
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for a program modifies the instructional procedures or 
changes the amount of student learning that takes place. 

If, from the point of view of the evaluator, the most 
relevant considerations are the decisions that will be 
made as a consequence of the information reporting, then 
it will be of utmost concern to determine the impact of 
describing objectives in behavioral terms. There is 
little evidence to substantiate that such descriptions, 
and the available data relating to them, modify the nature 
of the subsequent judgments by decision makers, 

A study by Eva Baker attempted to contrast the effect 
that behavioral and non-behavioral objectives have on pupil 
learning and found no significant differences in items 
directly measuring the objects. ves or on the transfer items. 
However, this study dealt with modification of student 
outputs as a function of using behavioral objectives, 
rather than the impact of such use on decision makers. 

In a study in which adult students are the decision makers, 
Blansy and McKie attempted to determine whether knowledge 
of instructional objectives in an adult education program 
assists participants in obtaining these objectives. The 
hypothesis that the group that was given behaviorally 
stated objectives would do significantly better than the 
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control group was confimed. However, in a typical 
educational situation, one ordinarily would think of the 
teacher or another intermediate control agent as the 
appropriate decision maker rather than the student, 

A study presently underway by Eva Baker at the UCLA Research 
and Development Center will attempt to examine the use 
of student response data (information related to the 
achievement of student behavioral objectives) in relation 
to the subsequent revisions made by the decision maker 
of the instructional material. 

Intuitive feeling, however, would lead to the view 
that, all things being equal, it is probably better to 
specify objectives than not to do so at all. With this 
in mind, and with a deep conviction at the Center for 
Study of Evaluation that the specification of system objec- 
tives should be the function of a local decision maker ra- 
ther than of an external body, the center is developing 
a system to help the decision maker determine such 
selections , 

In an attempt to provide local decision makers 
with behavioral objectives and appropriate test items, 
we have established an Instructional Objectives Exchange 
at the UCLA Center for the Study of Evaluation. 



- 24 - 



The exchange is under the direction of Rod Skager and 
Jim Popham and has been established in response to 
several problems presently existent in the field. These 
are: ( 1 ) the role of the teacher /decision maker as an 

objectives selector, rather than as an objectives gen- 
erator; (2) the need for test items related to objec- 
tives, and (3) the imminent duplication of efforts in 
various parts of the United States. 

While the Instructiona,! Objectives Exchange Project 
will function as a claringhouse in the area of objectives 
and items, our prime intended use of the exchange at the 
UCLA Research and Development Center goes beyond this. 

We plan to use some of the material collected in the Ex- 
change in order to study the form and use of behavioral 
objectives. For example we wan.t to answer the following 
questions: (l) Do alternative modes of stating objectives 

have a relationship to pupil performance? (2) Does using 
behavioral objectives as the basis for determining 
information requirements modify the nature of the ultimate 
judgments of decision makers? (3) What are the types 
of decisions made by teachers, administrators, etc., 
who have been presented with ol?jective-based data? 

We hope that the results of these studies will provide 
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some insights into the relevance of behavioral objectives 
as a part of the evaluation of relatively well defined 
instructional programs, particularly in the program 
development and program certification stages. 



A Response 



The activities of the Center for the Study of Evalua- 
tion at UCLA are vitally related to the evaluation problems 
faced by schools and school districts every day. We regard 
ourselves as a research and development unit whose goal is 
to ’’make a difference” in education. Our activities in 
conceptualizing evaluation are designed, among other things, 
to enable us to understand the potential relevance of var- 
ious procedures in evaluations of different types. Our 
School Evaluation project will hopefully provide insights 
into the information requirements of decision makers. 

The Instructional Objectives Exchange and Measurement 
System project will provide evidence as to the form and 
use of program objectives in decision making in the 
improvement and certification stages of evaluation. 

The kind of mapping of the domain that is exemplified 
by the three activities named will ultimately allow us to 
answer in some definitive way whether the need for speci- 
fication of objectives in evaluation is relevant or irrel- 
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